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Abstract — The concept of Public- key cryptosystem 
was innovated by McEliece's cryptosystem. The pub- 
lic key cryptosystem based on rank codes was pre- 
sented in 1991 by Gabidulin -Paramonov-Trejtakov 
(GPT). The use of rank codes in cryptographic 
applications is advantageous since it is practically 
impossible to utilize combinatoric decoding. This has 
enabled using public keys of a smaller size. Respective 
structural attacks against this system were proposed 
by Gibson and recently by Overbeck. Overbeck's 
attacks break many versions of the GPT cryptosystem 
and are turned out to be either polynomial or expo- 
nential depending on parameters of the cryptosystem. 
In this paper, we introduce a new approach, called the 
Smart approach, which is based on a proper choice of 
the distortion matrix X. The Smart approach allows 
for withstanding all known attacks even if the column 
scrambler matrix P over the base field F,. 

I. Introduction 

McEliece fTl has introduced the first code-based 
public-key cryptosystem (PKC). The system is 
based on Goppa codes in the Hamming metric, 
which is connected to the hardness of the general 
decoding problem. It is a strong cryptosystem but 
the size of a public key is too large (500 000 bits) 
for practical implementations to be efficient. 

Neiderreiter |2| has introduced a new PKC based 
on a family of Generalized Reed-Solomon codes; 
its public key size is less than the McEliece cryp- 
tosystem, but still large for practical application. 

Also, Gabidulin Paramonov and Trietakov have 
proposed a new public key cryptosystem, which 
is now called the GPT cryptosystem, based on 
rank error correcting codes in ([3], Q. The GPT 
cryptosystem has two advantages over McEliece's 
Cryptosystem. Firstly, it is more robust against 
decoding attacks than McEliece's Cryptosystem; 
secondly, the key size of the GPT is much smaller 
and more useful in terms of practical applications 
than McEliece's cryptosystem. 



Rank codes are well structured. Subsequently 
in a series of works, Gibson Q, Q developed 
attacks that break the GPT system for public keys 
of about 5 Kbits. The Gibson's attacks are efficient 
for practical values of parameters n < 30, where 
n is the length of rank code with the field F21V as 
an alphabet. 

Several proposals of the GPT PKC were 
introduced to withstand Gibson's attacks Q, 
18J. One proposal is to use a rectangular row 
scramble matrix instead of a square matrix. The 
proposal allows working with subcodes of the 
rank codes which have much more complicated 
structure. Another proposal exploits a modification 
of Maximum Rank Distance (MRD) codes where 
the concept of a column scramble matrix was 
also introduced. A new variant, called reducible 
rank codes, is also implemented to modify the 
GPT cryptosystem |9|, ifTOl . All these variants 
withstand Gibson's attack. 

Recently, R. Overbeck E], 112, and lUll has 

proposed new attacks, which are more effective 
than any of Gibson's attacks. His method is based 
on two factors : a) a column scrambler P that is 
defined over the base field , and b) the unsuitable 
choice of a distortion matrix X . However, Over- 
beck managed to break many instances of the GPT 
cryptosystem based on the general and developed 
ideas of Gibson. 

Kshevetskiy in |[T9l suggested a secure approach 
towards the choice of parameters for avoiding 
Overbeck's attacks based on suitable choice of 
the distortion matrix X. Independently, Loidreau 
in 1201 proposed similar method. Gabidulin |[T4l 
has offered a new approach called the Advanced 
approach, which makes the cryptographer define 
a proper column scrambler matrix over the ex- 



tension field without violating the standard mode 
of the PKC. The Advanced approach allows the 
decryption of the authorised party, and prevents an 
unauthorized party from breaking the system by 
means of any known attacks. The two approaches 
withstand Overbeck and Gibson's attacks. 

Recently, we have presented another variant of 
the GPT public key cryptosystem [211 . based on 
a proper choice of column scrambler matrix over 
the extension field. This variant, which we call the 
Instrumental approach, is secure against all known 
attacks. 

In this paper, we introduce a new approach 
called the Smart approach, which is based on a 
proper choice of the distortion matrix X . The 
Smart approach allows for withstanding all known 
attacks even if the column scrambler matrix P over 
the base field ¥q. 

The rest of this paper is structured as follows. 
Section 2 gives a short introduction to rank codes. 
Section 3 describes the GPT cryptosystems. Sec- 
tion 4 discusses the Overbeck's attacks. Section 5 
presents the Smart approach of GPT PKC cryp- 
tosystem with two examples. Finally, section 6 
concludes the paper with some remarks. 

II. Rank codes 
Let us introduce the basic notion of rank codes 
(|3], ifTSl . Let ¥q be a finite field of q elements 
and let F^n be an extension field of degree N. Let 
X = {xi,X2, ■ . ■ ,Xn) be a vector with coordinates 
in VgN. 

The Rank norm of x is defined as the maximal 
number of Xi, which are linearly independent over 
the base field and is denoted Rk(x | ¥q). 
Similarly, for a matrix M with entries in F^n , the 
columns rank is defined as the maximal number of 
columns, which are linearly independent over the 
base field Fg, and is denoted Rkcoi(M|Fg). 
We distinguish two ranks of the matrix: 

1) The usual rank of matrix M over F^w - 

Rk{M \¥qN). 

2) The column rank of a matrix AI over the base 
field F, - Rkcoi(A/ I F,). 

The column rank of the matrix M 
depends on the field. In particular, 
Rkcoi(M \¥g)> Rkcoi(M|F,«) 
The Rank distance between x and y is defined 
as the rank norm of the difference x — y: 
d(x,y) =Rkeoi(x-y | F,). 

Any linear (n, fc, d) code 6 C F'^V fulfils the 
Singleton-style bound lITSi for the rank distance: 

Nk<Nn-{d-l)ma.x{N,n}. (1) 



A code C reaching that bound is called a Maxi- 
mal Rank Distance (MRD) code. 

The theory of optimal MRD (Maximal Rank 
Distance) codes is given in [|l5j. 



The notation g[i] 



means the i-th 



Frobenius power of g. It allows to consider both 
positive and negative Frobenius powers i. 

For n < N, a generator matrix of a (n, k, d) 
MRD code is defined by a matrix of the following 
form: 



51 52 



gr, 



,[1] 



■51 



[fe-1] ^[fc-1] 



92 



[k- 

gii 



(2) 



where 51 , 52 , • • • , 5n are any set of elements of the 
extension field F^w which are linearly independent 
over the base field ¥q. 

A code with the generator matrix (|2]i is referred 
to as {n,k,d) code, where n is code length, k 
is the number of information symbols, d is code 
distance. For MRD codes, d = n — k + 1. Let 
m — (rrii, 7712, . . . , nik) be an information vector 
of dimension k. The corresponding code vector is 
the n-vector 

g(m) mGfc. 

If y = g(m) + e and Rk(e) = s < t ^ ^ , 
then the information vector m can be recovered 
uniquely from y by some decoding algorithm. 
There exist fast decoding algorithms for MRD 
codes ifTsll . llT6l . A decoding procedure requires 
elements of the (n — fc) x n parity check matrix H 
such that GfcH^ = 0. For decoding, the matrix H 
should be of the form 



H 



hi 



[1] 



[d-2] 



,[1] 



[d-2] 



hn 



(3) 



where elements hi,h2, ■ ■ ■ ,hn are in the extension 
field ¥qN and are linearly independent over the 
base field ¥q. 

The optimal code has the following design pa- 
rameters: code length n < N; dimension k = 
n — d + 1, rank code distance d = n — k + 1. 

III. The GPT Cryptosystem 

Description of the standard GPT cryptosystem. 

The GPT cryptosystem is described as follows: 
Plaintext: A Plaintext is any fc-vector m = 
(toi,TO2, • • • jTOfe), rus G F^N, s = 1, 2, . . . , fc. 
In previous works, different representations of the 



public key are given. All of them can be reduced 
to the following form. 

The Public key is a fc x (n + ii) generator matrix 



Gpub — S [X G/j] P. 



(4) 



Let us explain roles of the factors. 

• The main matrix is given by|2] It is used to 

correct rank errors. Errors of rank not greater 

than ^^Y^ can be corrected. 
« A matrix S is a row scrambler. This matrix is 

a non singular square matrix of order k over 

« A matrix X is a distortion (kxti) matrix over 
¥qN with full column rank Rkcoi(^ | Fq) 
h and rank Rk(X \ W^n) = tx , tx < h. 
The matrix [X G^] has full column rank 
Rkeoi([X Gk] I ¥g)=n + ti. 
m A matrix P is a square column scramble 

matrix of order {ti + n) over F^. 
m ti + n may be greater than N, but n < N. 
The Private keys are matrices S, G^, X, P sep- 
arately and (explicitly) a fast decoding algorithm 
of an MRD code. Note also, that the matrix X is 
not used to decrypt a ciphertext and can be deleted 
after calculating the Public key. 
Encryption: Let m = (toi, m2, . . . , rrifc) be a 
plaintext. The corresponding ciphertext is given by 

c = mGpub + e = mS [X G^] P + e, (5) 

where e is an artificial vector of errors of rank t2 
or less. It is assumed that ti + t2 < t — [-^^J 

Decryption: The legitimate receiver upon re- 
ceiving c calculates 

C = (c^, C2, . . . , Cf^_|_„) = 

cP-i=mS[X Gfc]+eP-i 
Then from c he extracts the subvector 

C = (Ct^ + i,q^+2:---:Ct^+„) =mSGfe+e , 

(6) 

where e is the subvector of eP ^ . Then the legit- 
imate receiver applies the fast decoding algorithm 
to correct the error e , extracts mS and recovers 
TO as m = (mS)S"^. 

In this system, the size of the public key is 
V = k{ti + n)N bits, and the information rate 
is 

ti+n 

IV. OvERBECK's Attack 

In ifTTI . lfT2l . and lfT3l . new attacks are proposed 
on the GPT PKC described in the form of g] It is 



claimed, that similar attacks can be proposed on all 
the variants of GPT PKC. 

We recall briefly this attack. 
We need some notations. 



be the Frobenius 



over ¥gN, let a{T) 



For X G ¥qN let (t{x) - 
automorphism. 
For the matrix T ~ [tij) 

HUj)) = (4)- 

For any integer s, let cr^(T) = aia^^^iT)). 

It is clear that = a. Thus the inverse exists 

(7-1 

The following simple properties if cr are useful: 

• a{a + b) = (7(a) + crib), 
m a{ab) — a{a)(j{b). 

m In general, for matrices (t(T) 7^ T. 

• If P is a matrix over the base field F^, then 
a(P) - P. 

Description of Overbeck's attack: To break a 
system, a cryptanalyst constructs from the public 
key Gpub = S [X G/j] P the extended public 
key Gcxt,pub as foUows: 



G 



cxt ,pub 



s 

a(S) 



X 

a(X) 



Gpub 
o"(Gpub) 

T"(Gpub) 

Gfc 
t(GO' 



a"(S) [a"(X) a"(G,)] 



(7) 



The property that ct(P) = P, if P is a matrix over 
the base field F,, is used in (|2|. 
Rewrite this matrix as 



'cxt .pub 



Xc 



Go 



(8) 



where 



Scxt — 


Diag [S 


a(S) ... 


a"(S)] 




r X 1 




Gfc 




a(X) 






Xcxt 


, Gcxt 








.a"(Gfc)- 



(9) 



Choose 



(10) 



For Si kxti matrix X, let Xi be the (fc — 1) x ti 
matrix, obtained from X by deleting the last row. 
Similarly, let X2 be the (fc— 1) xti matrix, obtained 
from X by deleting the first row. 



Define a linear mapping T : F 



kxti 



¥' 



(fc-l)xti 



by the rule: if X G 

a(Xi) -X2. Let 



then r(X) = Y = 

Yext = [Y <7(Y) <72(Y) ... a"-i(Y)]^ (11) 



Using this and other suitable transformations of 
rows, one can rewrite for analysis ^ and (|9]l in 
the form 



'pubjCxt 



z 




Yoxt 






(12) 



where G„_i is the generator matrix of the {n, n - 

1,2) MRD code. 
Let us try to find a solution u of the system 



z 







Pu^ 



0, 



(13) 



where u is a vector-row over the extension field 
¥„N of length ti + n. Represent the vector Pu^ 



' 9 

as 



Pu' 



where the subvector y has length ti and h has 
length n. Then the system ( fT3] ) is equivalent to the 
following system: 



Zy^ 



' = 0, 

Assume that the next condition is valid: 

Rk(Yea:t|F^«) =ii. 

Then the equation (fTsT i has only the trivial solution 
= 0. The equation ( fT4l i becomes 



(14) 
(15) 

(16) 



(17) 



It allows to find the first row of the parity check 
matrix for the code with the generator matrix ( fT2] l 
(see,|Tr|, fT2l, and fT3l, for details). Hence this 
solution breaks a GPT cryptosystem in polynomial 
time. The Overbeck's attack requires 0{{n + tif) 
operation over F^n since all the steps of the attack 
have at most cubic complexity on n + ti. 

V. Smart approach 

To withstand Overbeck's attack, the cryptogra- 
pher should choose the matrix X in such a manner 
that 

Rk(Ye^t I F^«) = ti -a, (18) 

where a > 2. In this case, the system ( fT5l ) has q"-^ 
solutions . Hence the exhaustive search over 
is needed. The work function has order 0{q°'^ (n+ 

tif ) and Overback's attack fails. 

One method to provide the condition ( fTSl ) is 
proposed in tl9il , 120 1. Choose the matrix X over 
the extension field F^n in such a manner that the 
following conditions are satisfied: 

ti = Rkcoi(X I F,) > 
rx = Rk(X|F,iv) = 



< k. 



(19) 



Overbeck's attack is exponential on a and has the 
minimum complexity at least O {(f^ {n + ti)"^). 

We propose an alternative Smart approach. The 
point is to choose the matrix X in such a man- 
ner that the corresponding matrix Y = r(X) 



has column rank Rk(Y | F^) not greater than 
ti — a, a > 2. 

The following result is evident. 

Lemma 1: If Rk(Y | F,) = s, then Rk(Yoxt | 

F,) = s. 

Corollary 1: Rk(Ycxt I F^n) < Rk(Ycxt | Fq) = 
s = Rk(Y I F,). 

a ) The simple case: Let a matrix X be of the 
following form: 



(20) 



Here m is a random vector over the extension field 
WgN with full column rank ti and vectors Si, i = 
1, . . . , k—1, are random vectors over the base field 
¥q such that the matrix 



m 




" ' 




+ 


Sl 






Sfe-l. 



[0 SI 



Sfc-l 



has rank ti — a. Then the matrix Y = T(X) has 
the form 



Sl - S2 



Sfe-1 - Sfc 



(21) 



This matrix is a matrix over the base field ¥q and 
has rank ti — a too. It follows that 



Hence 



(t(-si) 




-Sl 


0-(si - S2) 




Sl - S2 


_0-(Sfc_i - Sfc)_ 




.Sfc-l - Sfc_ 



(22) 





- Y 




"Y" 




a(Y) 




Y 




a"-i(Y). 




Y_ 



(23) 



Therefore Rk(Ye^t | F^n) = Rk(Y | Fqw) = 
ti — a, and the condition ( fTSl ) is satisfied. 

As in the previous case, the proposed Smart 
approach shows that Overbeck's attack is expo- 
nential on a and has the bit complexity at least 
(9((7"^(n + ti)3). 

It has been shown that the Smart approach 
presented above is secure against all known attacks 
including the recent attack presented by Overbeck 
in |13|. 

Example 1: Let n = 8, = 4, = 8, t = 5, ti = 
4, g = 2, a = 2 

Let the extension field F28 be defined by the primitive 
polynomial r{x) = 1 + x'^ + x'^ + x"^ + , and let a 
he a primitive element of the field. Choose the matrix 
X as in ( i20b . A vector m of full column rank ti = 4 is 



defined as m = \a 



Choose vectors 



Sl, S2, S3 as Sl = fl 1 Ol,S2=fl 



1 , 



S3 = fO 1 



Then we obtain 
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a4 




1 
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1 
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^20 + 
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a24 + i 


a* 4- 
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q40 




Q:48 + 1 


ai6 H 


- 1 





The corresponding matrix Y is as follows; 



110 
11 
110 



(24) 



(25) 



It has rank ti—a = 2. The attack is exponential on a and 
has the bit complexity at least 0(q''^ (n+tif) = 0(2^^ 
bite operations. 

b) The general case: Let X be a ma- 
trix consisting of a Frobenius-type columns and 
ti — a non-Frobenius columns. A column w is 
called Frobenius-type if it has the form w = 

(w wl^l ... wl'^^^l)^. It is clear that T(w) = 
0. Hence the matrix Y = ^(X) will have a all 
zero columns and column rank ti ~ a and by 
Corollary [T] the matrix Ycxt has rank not greater 
than ti—a. The result is valid also if suitable linear 
combinations of non-Frobenius columns are added 
to Frobenius-type columns. 

Example 2: In conditions of the previous example, let 
matrix X be as follows: 



X 



„24 I „12 



„10 



a 



The third column is added to the first Frobenius-type, 
and the fourth is added to the second Frobenius-type, so 
a = 2. Column rank of X is = 4. The corresponding 
matrix Y = r(X) is of the form: 











It has rank ti — a = 2. 

In general, Overbeck's attack fails when aN > 60. 

VI. Conclusion 

We have introduced the Smart approach as a technique 
of withstanding Overbeck's attack on the GPT Public key 
cryptosystem, which is based on rank codes. 

It is shown that proper choice of the distortion matrix 
X over the extension field F^jv allows the decryption 
by the authorized party and prevents the unauthorized 
party from breaking the system by means of any known 
attacks. 
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